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Developing  Temporal  Markers  to  Profile  Operational  Errors 


INTRODUCTION 

Researchers  and  operational  personnel  alike  generally 
accept  the  view  that  air  traffic  operational  errors1  (OEs), 
like  many  types  of  events  in  complex  systems,  evolve 
over  time  (e.g.,  Dekker,  2002;  Dorner,  1996;  Harry 
and  Schroeder,  2000;  Klein,  Orasanu,  Calderwood,  and 
Zsambok,  1993;  Perrow,  1999;  Petersen,  1996;  Phimis- 
ter,  Bier,  and  Kunreuther,  2004;  Reason,  1997;  Sagan, 
1993;  Salas  and  Klein,  2001).  If  we  generally  accept 
that  OEs  evolve  over  time,  then  we  can  expect  OEs  to 
have  temporal  characteristics.  However,  the  truism  that 
events  in  complex  systems  have  a  temporal  component 
has  not  yet  led  to  development  of  better  techniques  for 
profiling  the  temporal  progression  of  evolving  events  such 
as  OEs.  This  project  was  conducted  to  begin  to  fill  this 
gap.  Temporal  information  may  give  OE  analysts  another 
means  to  identify  and  evaluate  a  variety  of  potential  OE 
reduction  strategies. 

A  commonly  held  view  is  that  system  and  human 
vulnerabilities  can  converge  at  one  moment  or  may  be 
distributed  over  the  course  of  an  evolving  situation. 
In  either  case,  they  can  contribute  to,  or  cumulate  in, 
unsuccessful  outcomes.  Many  researchers  have  devoted 
attention  to  identifying  and  analyzing  these  precursor 
and  error-producing  human  and  system  conditions  (e.g. , 
Bier,  1 998;  Krause,  S.S.,  2003;  Petersen,  1996;Phimister, 
Bier,  and  Kunreuther,  2004;  Rasmussen,  1980;  Reason, 
1990,  1997).  On  the  other  hand,  the  study  of  temporal 
characteristics  in  this  context  has  received  little  attention 
and  is  generally  conducted  relative  to  its  perception  and 
related  subjective  experiences  (e.g.,  Block,  1990;  Gibbon 
and  Allan,  1984). 

Although  many  types  of  data  are  currently  reported 
after  an  OE,  by  identifying  the  temporal  characteristics 
of  OEs,  we  should  be  able  to  understand  OEs  from  a 
different  perspective.  Presently,  over  200  elements  of 
information  are  recorded  as  part  of  the  Federal  Aviation 
Administration’s  (FAA’s)  OE  reporting  process.  In  addition 
to  a  description  of  when  and  where  the  event  occurred, 

1  An  operational  error  is  defined  as  an  occurrence  attributable  to  an 
element  of  the  air  traffic  system  in  which  (1)  less  than  the  applicable 
separation  minima  results  between  two  or  more  aircraft,  or  between 
an  aircraft  and  terrain  or  obstacles  (e.g.,  operations  below  minimum 
vectoring  altitude  (MVA);  equipment  /  personnel  on  runways),  as 
required  by  FAA  Order  71 10.65  or  other  national  directive;  or  (2)  an 
aircraft  lands  or  departs  on  a  runway  closed  to  aircraft  operations  after 
receiving  air  traffic  authorization,  or  (3)  an  aircraft  lands  or  departs 
on  a  runway  closed  to  aircraft  operations,  at  an  uncontrolled  airport 
and  it  was  determined  that  a  NOTAM  regarding  the  runway  closure 
was  not  issued  to  the  pilot  as  required  (FAA  Order  7210.56,  2002). 


these  encompass  characteristics  of  the  controllers  and 
supervisors  involved,  training,  past  performance  ofperson- 
nel  (i.e.,  involvement  in  previous  incidents),  equipment 
and  aircraft.  Items  also  cover  data  posting  (e.g. ,  computer 
entries  and  flight  progress  strips),  controller’s  use  of  radar 
displays  to  maintain  knowledge  of  the  traffic  situation, 
communications,  coordination,  transfer  of  position 
responsibilities  (i.e.,  position  relief  briefings),  and  in  air 
traffic  control  towers,  the  observation  of  aircraft. 

A  chronology  (timeline)  of  the  OE  is  routinely  devel¬ 
oped  by  quality  assurance  personnel  at  air  traffic  facilities 
during  the  FAA’s  OE  reporting  process  as  part  of  the  Final 
Operational  Error  or  Deviation  Report  (Form  7210-3; 
FAA,  2002).  Directions  for  completing  the  Final  OE/D 
Report  are: 

Explain,  in  chronological  order,  each  factor  relevant  to 
the  incident.  Tell  a  detailed  story,  describing  the  pertinent 
actions  of  all  those  involved  (e.g.,  controllers  by  position, 
supervisors,  aircraft,  etc.).  It  should  be  apparent  what  ac¬ 
tions  (or  lack  of)  contributed  to  or  caused  the  incident. 
Include  any  explanations  necessary  from  previous  blocks. 

(Pg-  7) 

Two  examples  of  timelines  from  our  archive  are  shown 
below,  although  the  amount  of  content  and  format  of 
timelines  included  in  the  Final  Report  varies  widely 
across  facilities. 

Example  1; 

1356:24  DAL1906  checked  on  frequency  at  FL330. 

1358:42  R63  cleared  DAL1906,  due  to  ZID  Letter  of 

Agreement,  direct  Pocket  City,  then  MOSEY5 
arrival  to  CVG.  R63  cleared  DALI  906  to  turn  left 
heading  3-5-0,  when  able  direct  Pocket  City. 
1358:52  DAL1906  acknowledged  the  heading  of  3-5-0 

direct  Pocket  City  and  MOSEY. 

1401:16  AAL2736  advised  R63  that  his  TCAS  had  acti¬ 

vated  advising  him  to  climb  due  to  traffic  at  his 
altitude. 

Example  2: 

1 93 1 :35  When  DAL0025  reported  on  frequency  climbing 

to  17,000  feet,  Specialist  “A”  cleared  the  aircraft  to 
climb  and  maintain  FL220  and  to  proceed  direct 
Birmingham  (BHM). 
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1932:14  When  DAL1347  reported  on  frequency,  Special¬ 
ist  “A”  cleared  the  aircraft  to  climb  and  maintain 
17,000  feet. 

1933:12  Separation  was  lost. 

1933:20  Specialist  “A”  cleared  DAL1347  to  level  off  at 
present  altitude.  DAL  1347  did  not  respond  until 
the  third  call  at  which  time  the  aircraft  was  already 
level  at  17,000  feet. 

1933:37  Specialist  “A”  instructed  DAL0025  to  expedite 
climb  to  FL220. 

1933:47  Specialist  “A”  cleared  DAL  1347  to  descend  and 
maintain  16,000  feet. 

Various  stakeholder  groups  have  long  monitored  in¬ 
coming  OE  reports  so  that  strategies  could  be  developed 
to  reverse  human  performance  errors,  such  as  readback/ 
hearback  errors,  lack  of  positive  control  and  misapplication 
of  procedures.  Several  analyses  incorporated  a  temporal 
dimension.  For  example,  air  traffic  operations  tracked  the 
number  ofOEs  relative  to  the  controller’s  time  on  position 
and  number  of  aircraft  being  handled  by  the  controller 
(Pounds  and  Ferrante,  2005).  Researchers  examined 
whether  information  about  the  stabilization  of  control¬ 
lers’  scanning  patterns  during  the  first  1 0-20  minutes  on 
position  could  be  used  to  inform  our  understanding  of 
how  controllers  establish  situation  awareness  during  that 
time  (Smolensky,  1993).  Although  much  attention  has 
been  devoted  to  understanding  the  transfer  of  position 
responsibilities  and  despite  an  ongoing  interest  in  time  as 
a  factor  in  relation  to  various  air  traffic  operations,  there 
is  currently  no  way  to  identify  and  analyze  information 
about  how  an  OE  evolves  temporally. 

One  means  for  understanding  the  evolution  of  an 
OE  might  be  to  look  for  events  that  could  be  identified 
temporally,  which  all  OEs  have  in  common  or  which 
could  potentially  occur  in  every  OE.  We  called  these 
events  temporal  markers  (TMs).  These  points  may  differ 
in  when  they  occur  during  the  OE  timeline,  but  they 
occur  in  some  form  in  most,  if  not  all,  OEs.  TMs  could 
also  be  used  to  compute  time  intervals.  We  called  these 
time  intervals  calculated  between  two  TMs  the  Temporal 
Marker  Time  Spans  (TMTSs). 

This  idea  led  us  to  first  identify  an  exhaustive  list  of 
TMs.  The  list  was  then  organized  into  a  framework  and 
tested  using  a  convenience  sample  of  OEs  to  see  whether 
usingTMs  would  yield  new  information  about  OEs.  This 
report  details  how  this  was  accomplished,  the  resulting 
framework  of  TMs,  examples  of  new  information  and 
examples  of  difficult  questions  that  could  be  answered 
by  this  type  of  analysis. 


Step  1  —  Identifying  the  TM  Framework 

Method 

In  Step  1 ,  we  first  had  to  define  (operationalize)  what 
we  meant  by  temporal  markers  and  determine  whether  any 
events  in  OEs  matching  the  definition  could  be  identified 
using  available  materials  generated  from  previous  OE 
investigations.  In  Step  2,  we  used  the  list  of  temporal 
markers  identified  in  Step  1  to  analyze  a  convenience 
sample  of  OEs.  A  short  summary  completes  each  section, 
although  the  principal  discussion  of  the  results  is  reserved 
for  the  conclusion  of  the  paper. 

We  defined  a  temporal  marker  (TM)  as  an  objective  (or 
calculable)  point  in  time  that  can  be  identified  during  the 
unfolding  of  the  OE.  Temporal  markers  must  be  identifi¬ 
able  (or  calculated)  from  data  sources  available  in  the  AT 
facility.  The  range  of  possible  TMs  was  bounded  only  by 
“the  context  of  an  operational  error.”  These  event  points 
could  occur  at  different  times  during  an  OE,  but  they 
would  occur  in  some  form  in  most,  if  not  all,  OEs. 

Participants 

Three  retired  FAA  air  traffic  control  specialists  (ATCSs) 
with  experience  in  OE  investigations  were  recruited  as 
ATC  subject  matter  experts  (SMEs)  to  assist  in  the  devel¬ 
opment  of  a  comprehensive  list  of  TMs.  Their  collective 
experience  included  87  years  of  facility  management 
and  operational  supervision  in  en  route  and  terminal 
operations. 

Materials 

A  convenience  sample  of  six  OEs  from  one  Air  Route 
Traffic  Control  Center  (ARTCC)  was  used  for  Step  1. 
These  were  selected  because  both  the  Final  OE/D  Report 
(FAA  Form  721 0-3)  and  the  Systematic  AirTraffic  Opera¬ 
tion  Research  Initiative  (SATORI)  re-creation  for  each 
were  available  from  an  existing  archive.  SATORI  (Rodgers 
and  Duke,  1 993)  was  developed,  tested  and  fielded  to  all 
en  route  facilities  and  regional  quality  assurance  offices 
with  the  goal  of  gaining  “a  better  understanding  of  the 
interaction  between  the  various  elements  of  displayed 
information,  verbal  interactions  and  the  control  actions 
taken  by  air  traffic  control  specialists  ”  (pg.  1).  Information 
about  the  SATORI  system’s  capabilities  first  appeared  in 
ATCA’s  Journal  ofAir  Traffic  Control  (Rogers  and  Duke, 
1993).  SATORI  is  still  currently  in  operational  use  and 
enables  its  users  to  re-create  segments  of  operational  traffic 
in  a  format  similar  to  what  was  displayed  to  the  ATCS,  for 
example,  showing  relative  location  and  separation,  speeds 
and  headings  of  aircraft.  Among  other  things,  SATORI 
can  display  full  and  limited  data  blocks,  beacon  targets 
and  conflict  alerts.  Video  and  audio  are  synchronized  and 
the  air  traffic  situation  can  be  displayed  in  four  dimen¬ 
sions.  At  en  route  facilities,  SATORI  systems  enable  the 
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facility  quality  assurance  staff  to  re-create  OE  situations 
for  the  controllers  involved. 

The  SMEs  were  provided  with  our  definition  of  a 
temporal  marker  and  a  list  of  some  TM  examples,  such 
as  the  time  when  the  aircraft  entered  the  controller’s  as¬ 
signed  airspace,  the  time  when  the  aircraft  checked  on  the 
controller’s  frequency  and  the  time  when  the  controller 
issued  the  first  control  instruction  to  the  pilot. 

Procedure 

The  SMEs  convened  as  a  group  on  several  occasions 
to  create  an  exhaustive  list  of  TMs.  An  iterative  process 
was  used  to  develop,  test  and  modify  the  list.  At  the  first 
meeting,  the  principal  investigator  for  the  proj  ect  discussed 
the  project  goals  with  the  SMEs.  The  SMEs  then  observed 
SATORI  re-creations  and  reviewed  the  final  reports  from 
three  OEs  to  familiarize  themselves  with  the  materials 
for  the  project.  They  were  then  asked  to  generate  a  list 
of  events  that,  from  their  experience,  were  common  to 
all  OEs.  From  these,  the  SMEs  then  drafted  their  first 
TM  list  using  the  TM  definition. 

At  the  second  meeting,  the  list  was  reviewed,  discussed 
and  edited.  Once  this  draft  list  was  judged  by  the  SMEs 
to  be  complete,  it  was  tested  using  the  three  remaining 
OEs.  To  test  the  list,  the  SMEs  first  viewed  the  SATORI 
playbacks  for  the  three  OEs  and  noted  any  TMs  on  the 
list  that  needed  to  be  modified  or  any  that  needed  to  be 
added  to  or  deleted  from  the  list.  The  group  convened  a 
third  time  to  review  and  finalize  the  list. 

Results 

The  list  of  TMs  that  the  SMEs  developed  and  con¬ 
curred  with  included  identifiable,  objective  event  points 
that  occur  during  the  evolution  of  an  OE,  reflecting 
several  types  of  activities  and  demonstrating  what  kinds 
of  information  can  be  extracted  from  data  available  at 
facilities.  TMs  enable  OEs  to  be  described  in  more  detail 
than  is  generally  the  case. 

Table  1  shows  the  final  list  organized  as  a  general 
framework.  Columns  in  the  table  represent  the  number¬ 
ing  system  (TM  #),  TM  definitions2,  sources  where  a 
time  stamp  for  the  TM  can  be  found  (Data  Source)  and 
one  or  more  columns  for  the  aircraft  of  interest  (A/C  #). 
Column  headings  in  Table  1  show  how  the  analyst  would 
set  up  the  framework  to  examine  TMs  in  an  OE  with 
two  aircraft  in  conflict  and  another  aircraft  of  interest. 
Each  column  is  discussed  below  as  it  would  be  used  by 
an  analyst. 


2  Throughout  the  framework,  controllers  are  described  as  appropriate 
according  to  their  relationship  with  each  other  or  with  the  aircraft,  i.e. , 
receiving,  relieving,  transferring.  This  helps  the  analyst  to  remember 
relative  relationships  in  the  OE. 


Aircraft  Numbering  Methodology 

The  analyst  first  numbers  each  aircraft  to  permit  TMs 
to  be  identified  for  each  aircraft  as  needed.  Aircraft  are 
designated  systematically  as  A/Cl,  A/C2,  A/C3  and  so 
on.  The  number  assigned  to  the  aircraft  is  based  on  the 
sequence  of  its  entry  into  the  controller’s  airspace  where 
the  OE  occurred. — whether  by  (a)  coordination  accom¬ 
plished  between  controllers  using  the  land  line,  (b)  using 
automated  handoff  and  acceptance,  or  (c)  pilot  initiated 
contact  with  ATC.  The  first  aircraft  discussed  (or  flashed) 
between  the  handoff  and  receiving  controllers,  or  that 
contacted  the  controller  directly  (e.g.,  for  VFR  service 
in  the  airspace),  is  labeled  as  A/Cl.  The  second  aircraft 
discussed  (or  flashed)  between  the  handoff  and  receiving 
controllers,  or  that  contacted  the  controller  directly,  is 
labeled  as  A/C2  and  so  on.  All  aircraft  of  interest  as  ele¬ 
ments  of  the  OE  situation  are  assigned  a  number  using 
this  method.  In  addition  to  an  aircraft  that  was  being 
transferred  between  handoff  and  receiving  controllers,  an 
aircraft  might  also  have  contacted  the  controller  directly, 
for  example,  for  VFR  service  through  the  airspace,  to 
request  flight  following,  or  to  request  an  IFR  clearance 
after  departing  an  uncontrolled  airport  where  the  center 
provides  departure  services. 

An  example  of  how  aircraft  are  numbered  is  as  follows. 
Consider  a  situation  in  which  an  aircraft  (Cessnal234) 
departed  an  uncontrolled  airport.  This  airport  did  not 
have  an  affiliated  Terminal  Radar  Approach  Control 
(TRACON)  facility  so  ATC  services  for  pilots  arriving 
and  departing  this  airport  were  provided  by  the  center 
controller  working  the  center’s  Alpha  sector. 

•  The  pilot  of  Cessna  1234  contacted  the  Alpha  sector 
controller  at  0800  and  requested  IFR  clearance  through 
Alpha  sector. 

•  Alpha  sector  controller  issued  the  pilot  of  Cessna  1234 
the  appropriate  clearances. 

•  The  Alpha  sector  controller  also  verbally  accepted 
a  handoff  of  another  aircraft  (MEL0098)  from  an 
adjoining  sector  controller  at  0805. 

•  Another  aircraft  (E1RZ990)  was  flashed  to  the  Alpha 
sector  controller,  who  then  clicked  on  the  flashing 
datablock  and  accepted  the  handoff  of  E1RZ990  at 
0806. 

•  The  pilot  of  HRZ990  requested  to  remain  in  level 
flight  at  35,000  feet  (to  transition  Alpha  sector). 

•  Cessnal234  and  MEL0098  lost  separation  at  081 1. 

•  To  reestablish  separation  between  Cessnal234  and 
MEL0098,  the  Alpha  sector  controller  instructed 
MEL0098  to  climb  to  35,000  feet. 

•  At  0812,  separation  was  lost  between  E1RZ990  and 
MEL0098. 
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Table  1.  Framework  of  temporal  markers  for  an  OE  situation  having  three  aircraft  of  interest,  including 
clarification  for  some  temporal  markers  (in  italics). 


Temporal  Markers  for  Center  and  Terminal  (Radar) 


Source 


Time  position  relief  briefing  completed. 


2  Time  relieving  controller  assumed  position. 


3  Time  controller  being  relieved  left  position. 


*  TM#1:  The  time  when  completion  is  acknowledged  by  the  receiving  controller. 

*  TM#3:  Position  is  considered  “left”  at  the  time  the  relieved  controller  unplugged  from  the  position. 


.  ,  Time  of  1st  Inter/Intra  facility  coordination  initiated  on  the  involved 
4->  A/C. 


.  . ,  Time  of  1st  Inter/Intra  facility  coordination  completed  on  the  involved 
-  A/C. 


*  TM#4a  and  4b:  land  line  coordination. 


A/C  A/C  A/C 
1  2  3 


5a 

Time  radar  identification  initiated  by  transferring  controller  (Handoff, 
Point  out)*. 

6,1,4 

■ 

5b 

Time  radar  identification  completed  by  receiving  controller  (Handoff, 
Point  out)*. 

6,1,4 

■ 

*  TM#  5a  and  5b:  These  two  TMs  may  be  the  same  controller  if  initial  identification  is  being  established. 

*  TM#  5a  and  5b:  If  no  landline  coordination  takes  place,  this  would  be  the  first  coordination  for  the  aircraft 
completed  electronically. 


6  Time  A/C  checked  in  on  receiving  controller's  frequency. 


^  Time  two-way  communications  established  between  pilot  and 

_ receiving  controller. _ 


8  Time  A/C  entered  receiving  controller's  airspace. 


a  Time  of  1st  ATC  instruction  by  receiving  controller. 


*  TM#  9  la:  may  occur  at  the  same  time  as  TM#7. 


g  lb  Time  pilot  verbally  responded  to  1st  ATC  instruction  from  receiving 
-  controller. 


1,2, 6, 9 


1  q  Time  of  first  point  where  positive  separation  between  A/C  was  not 
ensured. 


*TM#10  is  the  first  time  when  the  aircraft  were  set  on  conflicting  headings.  The  “first  point”  was  defined 
irrespective  of  the  location  of  the  aircraft  — 

1.  the  aircraft  could  be  in  the  previous  sector,  e.g.,  if  separation  was  not  guaranteed  before  the  aircraft 
was  “shipped”  to  the  next  controller. 

2.  the  aircraft  could  be  in  the  receiving  controller ’s  sector  after  the  controller  takes  the  handoff  of  the 
aircraft. 

This  can  occur  as  a  result  of  either  for  example)  - 

1.  controller  action  or  inaction  to  ensure  separation 

2.  pilot  actions  or  inactions  (e.g.,  on  wrong  frequency  in  new  sector) 


1 1  ^  Time  of  last  ATC  instruction  by  receiving  controller  prior  to  loss  of 
_ separation. _ 


1  lb  Time  pilot  acknowledged  last  ATC  control  instruction  by  receiving 
controller  prior  to  loss  of  separation. 


12  Time  conflict  alert  activated. 


Time  of  controller's  1st  instruction  to  avoid  separation  error. 
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Table  1  (continued). 


1 3_lb 

Time  pilot  verbally  responded  to  controller's  1st  instruction  to  avoid 
separation  error. 

4 

14 

Time  separation  lost. 

7,2,1 

151  a 

Time  of  controller's  1st  attempt  to  reestablish  separation. 

4 

1 5_lb 

Time  pilot  verbally  responded  to  controller's  1st  attempt  to  reestablish 
separation. 

4 

16 

Time  separation  reestablished. 

2,1 

a.  Left  column  shows  sequencing  codes.  Shading  indicates  element  groups. 

b.  Data  source  codes  are  listed  with  the  relevant  TM.  Data  sources  are  listed  in  Table  2. 


Table  2.  Sources  of  data  to  find  times  to  complete  the  Temporal  Markers  in  Table  1 . 


Code 

Data  Source 

1 

SATORI  or  RAPTOR 

2 

NTAP  (National  Track  Analysis  Program) 

3 

CDRs  (Continuous  Data  Recording) 

4 

Audio  tapes 

5 

SISO  (sign  in-sign  off  /out  records) 

6 

DART  (Data  Analysis  Reduction  Tool) 

7 

OEDP  (Operations  Error  Detection  Program,  also  known  as  the  “snitch  patch”) 

8 

Personnel  interviews 

9 

Flight  progress  strips 

To  maintain  sequence  information,  Cessnal234  would 
be  designated  as  A/C  1  and  MEL0098  would  be  designated 
as  A/C2.  In  cases  where  a  third  aircraft  (e.g.,  HRZ990) 
influenced  the  events  leading  to  an  OE,  it  would  be 
designated  as  A/C3,  and  TMs  would  be  identified  for  it 
also  if,  for  example,  the  analyst  wanted  to  specify  its  role 
in  the  event.  In  this  example,  A/C3  lost  separation  with 
A/C2  subsequent  to  when  separation  was  lost  between 
A/Cl  and  A  /  C2  (e.g. ,  the  controller  created  a  second  OE 
while  reestablishing  separation  after  the  first  OE). 

In  another  case,  perhaps  A/C3  required  special  han¬ 
dling  or  multiple  communications  with  the  Alpha  sector 
controller  prior  to  when  separation  was  lost  between 
A/Cl  and  A/C2.  Examining  TMs  for  an  A/C3  or  A/C4 
would  shed  light  on  situations  where  other  aircraft  were 
involved  but  did  not  lose  separation  or  were  suspected 
of  being  a  contributing  factor  to  the  OE.  This  method 
of  examining  traffic  events  could  be  used  with  as  many 
aircraft  as  the  situation  warranted. 

No  explicit  rules  were  developed  during  this  project 
about  what  circumstances  dictate  that  the  analyst  should 
use  more  than  two  aircraft.  A  useful  guide  for  this  decision 
would  be  to  include  other  aircraft  when  the  information 
would  contribute  to  a  better  understanding  of  the  OE. 


Columns  as  many  A/C  as  needed  would  be  used  by  the 
analyst,  and  times  corresponding  to  each  TM  for  the  A / C 
would  be  entered  in  the  appropriate  column.  Using  this 
method,  the  framework  permits  more  than  two  aircraft 
to  be  tracked  if  the  analyst  determines  that  information 
about  the  third  aircraft  is  needed  to  better  understand 
how  the  OE  developed. 

Temporal  Markers  for  Center  and  Terminal  (Radar) 

This  column  of  the  framework  describes  each  TM 
identified  by  our  SMEs.  Several  of  the  TMs  are  anno¬ 
tated  for  clarification.  Each  annotation  is  shown  in  italics 
beneath  the  TM  to  which  it  refers. 

Data  Source 

The  SMEs  also  identified  the  source(s)  of  the  data 
needed  to  identify  each  TM  and  these  are  listed  in  the 
adjacent  Data  Source  column.  These  codes  represent 
the  sources  where  one  would  find  the  information  to 
identify  when  that  particular  TM  occurred.  The  data 
sources  corresponding  to  the  codes  are  listed  in  Table  2. 
Only  one  TM  requires  that  the  data  be  obtained  from 
facility  personnel  (TM#3:  “Time  controller  being  relieved 
left  position.”).  The  rest  are  obtained  using  data  from 
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automation  and  tools  used  in  en  route  facilities,  such 
as  the  National  Track  Analysis  Program  (NTAP),  flight 
progress  strips  and  audio  tapes. 

TM# 

The  left-hand  column  inTable  1  lists  the  alphanumeric 
code  assigned  to  each  type  ofTM.  Because  the  primary 
goal  ofthis  projectwas  to  identify  temporal  characteristics 
in  an  OE  as  it  evolved  over  time,  we  first  assigned  each 
type  ofTM  a  sequencing  code.  (The  obvious  choice  for 
a  numbering  method  was  to  use  the  common  method  of 
assigning  the  first  TM  as  #1,  the  second  TM  as  #2  and 
so  on.)  This  is  the  first  character  of  the  TM#  code  and 
indicates  the  relative  order  for  that  type  ofTM  within  the 
general  sequence  of  TMs  that  make  up  the  framework. 

Thus,  it  was  reasonable  to  assign  the  first  TM  of  the 
framework  to  the  first  moment  when  a  controller  assumed 
position  responsibility.  However,  transfer  of  position  is  a 
complex,  multi-step  procedure  and  encompasses  several 
sub-tasks,  so  these  steps  were  designated  asTM#  1 ,  TM#2 
and  TM#3  as  follows. 

•  TM#1:  Time  verbal  position  relief  briefing  completed. 

The  time  when  the  verbal  briefing  was  completed  was 

defined  as  the  moment  when  the  receiving  controller 
acknowledged  that  it  was  completed  (FAA,  2004).  By 
that  time,  the  relieving  controller  has  asked  any  ques¬ 
tions  necessary  to  ensure  a  complete  understanding  of 
the  operational  situation  and  these  have  been  completely 
answered  by  the  controller  being  relieved. 

•  TM#2:  Time  relieving  controller  assumed  position. 

The  time  when  the  relieving  controller  takes  responsi¬ 
bility  for  the  position  was  defined  as  (a)  when  the  relieving 
controller  makes  a  statement  or  otherwise  indicates  to 
the  controller  being  relieved  that  position  responsibility 
has  been  assumed  and  (b)  the  controller  being  relieved 
releases  the  position  (FAA,  2004). 

•  TM#3:  Time  controller  being  relieved  left  position. 

The  time  when  the  controller  left  position  was  defined 

as  the  time  when  the  controller  unplugged  his/her  head¬ 
set  from  the  position.  Generally,  the  time  at  which  the 
controller  is  relieved  of  position  responsibility  is  formally 
defined  as  the  time  when  the  controller  being  relieved  (a) 
signs  off  the  position  in  accordance  with  existing  direc¬ 
tives  or  (b)  otherwise  indicates  that  the  relief  process  is 
complete  (FAA,  2004).  In  some  situations,  the  controller 
being  relieved  may  remain  for  a  brief  period  and  continue 
to  observe  and  assist  with  traffic  demands.  During  certain 
times,  such  as  those  of  high  workload  or  complex  traffic 
situations,  the  controller  being  relieved  sometimes  stays 
at  the  position  to  assist  the  relieving  controller  after  the 


transfer  of  position  responsibility  has  been  accomplished. 
This  overlap  can  be  mandated,  encouraged,  or  dispensed 
with,  depending  upon  the  facility  or  circumstances. 
Therefore,  for  our  definition  ofTM#3,  we  considered  the 
relieved  controller  to  have  left  the  position  when  he/she 
unplugged  his/her  headset  from  the  position. 

One  number  is  not  sufficient  to  distinguish  all  of 
the  types  of  TMs  which  were  identified.  Some  TMs 
tend  to  occur  together  and  certain  types  of  TMs  tend 
to  be  repeated  throughout  the  course  of  an  OE.  Thus, 
additional  codes  were  needed  for  each  TM  and  so  two 
additional  codes  were  systematically  assigned  following 
an  underscore  after  theTM  sequence  number.  We  put  an 
underscore  after  the  first  number  to  distinguish  it  from 
the  next  characters  in  the  TM#  string  if  that  character 
was  also  a  number. 

The  numbers  and  letters  after  the  underscore  indicate 
which  TMs  in  an  OE  are  repeated  and  which  are  related. 
Repeated  instances  of  a  TM  are  signified  by  adding  a 
number  series  (i.e.,  “1,”  “2,”  “3,”  and  so  on)  after  the 
underscore  to  indicate  the TM’s  sequence  relative  to  that 
type.  Thus  the  “  1  ”  after  the  underscore  represents  the  first 
specific  instance  of  thatTM  type.  A  “2”  in  that  position 
after  the  underscore  represents  the  second  instance  of 
that  TM  type. 

The  second  character  after  the  underscore  is  a  letter. 
Letters  were  used  to  signify  one  element  of  a  TM  pair 
when  twoTM  types  are  sequential,  related  activities.  The 
letters  “a”  and  “b”  are  used  to  indicate  that  two  TMs  tend 
to  occur  coupled,  such  as  when  an  activity  is  initiated 
(i.e.,  “a”);  its  completion  is  also  included  (i.e.,  “b”).  That 
is,  if  one  occurs  the  other  will  also  usually  occur  as  shown 
in  the  example  below. 


TM# 

Temporal  Markers  for  Center  and  Terminal 
(Radar) 

4_la 

Time  of  1st  Inter/Intra  facility  coordination  initiated 
on  the  involved  A/C. 

4_lb 

Time  of  1st  Inter/Intra  facility  coordination 
completed  on  the  involved  A/C. 

The  shaded  rows  in  Table  1  highlight  the  TMs  that 
typically  occur  together,  either  as  groups  ofTM  sequences 
(i.e.,  TM#1,  TM#2,  TM#3)  or  as  pairs,  shown  by  the 
“a”  and  “b”  (e.g..,  TM#4_la,  TM#4_lb).  Using  TM# 
as  an  example,  the  “4”  indicates  that  this  TM  is  fourth 
in  the  TM  framework’s  sequence.  The  “  1  ”  following  the 
underscore  indicates  that  this  is  the  first  occurrence  of 
this  pair.  The  “a”  and  “b”  indicate  that  this  is  a  pair  of 
TMs  that  can  be  expected  to  occur  together.  These  are 
collectively  interpreted  as  the  first  occurrence  oftheTM#4 
pair.  This  coding  scheme  is  useful  when  some  types  of 
TMs  can  clearly  be  grouped  (e.g.,  the  position  relief)  or 
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Discussion 


separated  into  specific  elements  (e.g.,  a  controller-pilot 
communication  exchange).  For  example,  a  clearance  is¬ 
sued  by  the  receiving  controller  to  the  pilot  can  be  divided 
into  two  elements:  the  controller’s  communication  and 
the  pilot’s  response.  Coding  for  this  event  is  shown  by 
TM#9,  which  indicates  that  this  TM  pair  is  the  9th  type 
ofTM  in  the  framework,  the  first  occurrence  of  this  type 
(controller-pilot  communication) .  The  “a-b”  designation 
distinguishes  (a)  the  controller’s  instruction  to  the  pilot 
from  (b)  the  pilot’s  response  to  the  controller. 

1.  Time  of  1st  ATC  instruction  by  receiving  controller 
(TM  9_la). 

2.  Time  pilot  verbally  responded  to  1st  ATC  instruction 

from  receiving  controller  (TM  9 _ lb). 

As  described  above,  some  TMs  may  occur  once  for  an 
OE  (e.g.,  TM#l-3;  the  relieving  controller  took  position 
responsibility  after  a  briefing  from  the  controller  being 
relieved).  Other  types  can  be  repeated  (e.g.,  Time  of  1st/ 
2nd/  3rd/etc.  controller-pilot  communications)  and,  thus, 
instances  of  a  type  may  be  repeated  during  the  course  of 
an  OE’s  evolution.  All  TMs  that  are  relative  to  aircraft  can 
apply  to  each  aircraft  of  interest.  Therefore,  no  character 
was  included  in  the  temporal  marker  code  to  specify  in¬ 
dividual  aircraft.  The  aircraft  columns  in  the  framework 
are  used  to  distinguish  TMs  for  the  separate  aircraft. 

Although  listing  every  communication  with  the 
involved  aircraft  might  at  first  seem  unwieldy  or  unnec¬ 
essary,  the  cost-benefit  tradeoff  favors  including  them. 
The  current  requirement  (FAA,  2002)  for  examining  the 
unfolding  of  an  OE  is  to  examine  the  interval  beginning 
five  minutes  prior  to  the  involved  aircrafts’  initial  contact 
with  the  controller  having  the  OE  (in  our  example,  the 
Alpha  sector  controller)  and  concluding  five  minutes 
after  separation  is  lost  between  the  two  aircraft  in  the 
OE.  Gathering  the  data  and  transcribing  the  voice  tapes 
to  obtain  a  timeline  of  the  event  is  work  intensive  for  the 
facility  staff.  One  benefit  from  including  all  communi¬ 
cations  with  the  aircraft  in  the  TM  framework  is  that 
the  TM  structure,  by  providing  a  standardized  format 
for  the  timeline  (which  must  be  transcribed  anyway), 
would  provide  elaborative  information  about  exchanges 
and  activities  between  the  controller  and  the  aircraft 
involved  with  the  OE  beyond  that  which  is  currently 
available.  This  would  be  valuable  both  for  analysis  of  the 
individual  OE  case  and  when  summarized  at  facility-  or 
system-wide  levels. 


Using  TMs 

Once  the  TMs  in  the  template  are  populated  with 
data  from  the  OE,  sets  of  TMs  can  be  extracted  from 
the  framework  to  examine  specific  types  of  situations  and 
activities.  For  example,  our  SMEs  pointed  out  that  there 
is  a  point  in  every  OE  when  separation  is  not  ensured  for 
two  aircraft,  and  if  nothing  interceded  to  change  their 
trajectories,  they  would  eventually  lose  separation.  These 
two  concepts  need  to  be  clearly  distinguished:  (1)  when 
separation  is  not  ensured  versus  (2)  when  separation  is 
lost.  They  might  seem  similar  in  meaning  but,  in  reality, 
they  can  differ  markedly.  The  first,  “when  separation  is 
not  ensured,”  means  that  the  aircraft  are  on  courses  that 
will  lead  to  an  eventual  loss  of  separation  if  nothing  else 
happens  in  the  meantime,  such  as  another  action  by  a 
controller  or  a  pilot.  Two  aircraft  can  be  assigned  headings 
and / or  altitudes  that  will  eventually  result  in  an  OE  several 
minutes  (or  miles)  before  the  OE.  Thus,  events  might 
be  set  in  motion,  and  an  OE  could  essentially  begin  to 
develop  in  another  sector  prior  to  the  one  where  it  finally 
takes  place.  On  the  other  hand,  “loss  of  separation”  means 
that  the  aircraft  were  already  too  close  at  the  time  and 
were  in  a  conflict  situation  where  less  than  minimum 
separation  between  them  existed.  Thus,  “separation  is 
not  ensured”  can  be  thought  of  as  the  projection  of  the 
current  traffic  situation  into  the  future,  whereas  “loss  of 
separation”  is  the  outcome. 

Analysts  might  want  to  look  upstream  from  when  the 
OE  occurred  to  see  whether  earlier  actions  could  be  taken 
to  avoid  the  OE.  Because  TMs  can  be  uniquely  assigned 
to  each  aircraft  and  because  TMs  are  independent  of  the 
controller’s  airspace  boundaries,  this  TM  could  occur 
before  one  or  both  aircraft  are  in  the  receiving  controller’s 
airspace  (where  the  OE  occurred),  even  though  they  may 
have  already  been  handed  off  to  the  receiving  controller 
(who  “got  the  deal”)  with  aTM  sequence  as  follows. 

1.  TM#10:  Time  of  first  point  where  positive  separa¬ 
tion  between  A/C  (e.g.,  A/Cl  and  A/C2)  was  not 

ensured. 

2.  TM#12:  Time  conflict  alert  activated. 

3.  TM#14:  Time  separation  lost. 

4.  TM#16:  Time  separation  reestablished. 

Depending  upon  when  TM#10  occurred,  different 
strategies  for  mitigating  future  OEs  of  the  same  type  are 
revealed.  For  example,  controllers  are  expected  to  dem¬ 
onstrate  good  control  judgment  and  to  maintain  aware¬ 
ness  and  separation  of  traffic.  The  controller  using  good 
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control  judgment  would  not  place  aircraft  in  hand-off 
status  in  conflict  or  in  a  situation  leading  to  possible  loss 
of  separation  and  would  ensure  that  separation  standards 
are  maintained  at  all  times.  This  includes  ensuring  that 
aircraft  entering  and  departing  his/her  sector  airspace  are 
not  in  conflict  or  about  to  lose  separation.  The  controller 
would  also  use  procedures  that  do  not  place  workload  and 
stress  on  other  controllers  or  facilities.  Consider  the  dif¬ 
ferent  mitigating  strategies  suggested  by  two  situations. 

Variation  A  -TM#10  occurred  prior  to  the  handoff 
The  controller  is  required  to  “transfer  control  of  an 
aircraft  only  after  eliminating  any  potential  conflict 
with  other  aircraft  for  which  you  have  separation  re¬ 
sponsibility”  (FAA Order  71 10.65, 2004,  Para2-1-15), 
“[c  Insure  that,  prior  to  transferring  communications: 
potential  violations  of  adjacent  airspace  and  potential 
conflicts  between  aircraft  in  their  own  area  of  jurisdiction 
are  resolved”  (FAA,  2004,  Para  5-4-5)  and  “[rjestrictions 
issued  to  ensure  separation  are  passed  to  the  receiving 
controller”  (FAA,  2004,  Para  5-4-5). 

Ffowever,  this  only  implies  that  potential  conflicts 
between  aircraft  in  the  receiving  area  of  jurisdiction  are 
resolved  prior  to  transferring  communications.  In  other 
words,  the  transferring  controller  should  not  have  aircraft 
established  on  a  conflicting  course  when  transferring  them 
to  the  receiving  controller.  If  a  controller  were  to  transfer 
aircraft  to  another  sector  controller  without  ensuring  that 
their  separation  would  still  be  established  for  the  receiving 
controller,  the  aircraft  would  be  on  a  conflicting  course  but 
not  in  conflict  when  handed  off.  One  might  say,  in  other 
words,  that  an  OE  would  be  a  “set  up”  for  the  receiving 
controller,  who  would  receive  a  “dirty”  traffic  situation. 
In  these  cases,  the  receiving  controller  inherits  a  potential 
OE  situation  that  was  set-up  by  the  previous  controller 
and  ultimately  may  “get  a  deal”  unless  he/she  realizes 
after  taking  the  handoff  that  the  aircraft  are  on  a  con¬ 
flicting  course  and  takes  steps  to  ensure  their  separation. 
Mitigation,  in  these  cases,  could  target  the  procedures, 
controller  performance  and  other  factors  related  to  the 
controller  who  handed  off  the  aircraft,  rather  than  (or 
in  addition  to)  those  of  the  controller  in  whose  airspace 
the  OE  occurred. 

Variation  B  -TM#10  occurred  after  the  handoff 
After  receiving  aircraft,  the  controller  is  required  to 
maintain  appropriate  separation  standards  between  them 
(FAA,  2004).  When  a  controller  accepts  the  handoff  of 
aircraft  that  have  separation  ensured,  the  traffic  situation 
is  handed  off  “clean,”  and  the  receiving  controller  must 
maintain  separation  standards  between  them.  If  the  air¬ 
craft  are  then  set  on  a  conflicting  course  (either  by  the 
controller’s  action  or  inaction)  and  lose  separation,  then 


the  receiving  controller  still  “has  the  deal.”  However,  in 
this  case,  the  aircraft  were  not  on  a  conflicting  course 
or  in  conflict  when  handed  off  but  they  lost  separation 
while  in  the  receiving  controller’s  airspace.  In  other  words, 
the  situation  was  handed  off  “clean,”  but  the  receiving 
controller  made  it  “dirty.” 

To  better  understand  OEs,  analysts  would  like  to 
know  more  about  how  events  unfold,  e.g.,  how  often 
do  certain  types  of  events  happen  and  does  one  type 
make  an  OE  more  likely  to  occur  than  another?  This 
temporal  knowledge  could  then  be  combined  with  other 
information  (such  as  aircraft  types,  weather,  procedures) 
to  identify  strategies  to  improve  the  system. 

Summary  of  Step  1 

The  TM  framework  developed  in  Step  1  provides  a 
method  to  structure  an  OE.  By  using  objective  temporal 
information  to  populate  the  framework,  the  result  then 
can  be  used  in  different  ways,  depending  upon  the  issue 
of  interest.  For  example,  in  a  case  analysis  of  a  single  OE 
the  framework  could  be  used  to  clarify  temporal  relation¬ 
ships  between  a  number  of  aircraft  and  the  controller. 
Further,  the  organizing  structure  of  the  TM  framework 
permits  analysts  to  compare  TMs  across  many  OEs  to 
identify  possible  systemic  concerns. 

In  addition  to  identifying  temporal  markers,  the 
sequenced  nature  of  the  framework  could  also  be  used 
to  calculate  intervals  between  TMs  (a  Temporal  Marker 
Time  Span;  TMTS)  for  other  patterns  of  interest,  such  as 
the  TMTS  between  first  point  where  positive  separation 
was  not  ensured  and  time  that  separation  was  lost.  With 
this  in  mind,  the  framework  was  tested  to  determine 
whether  calculating  these  intervals  would  yield  additional 
information. 

Step  2  —  Testing  the  TM  Profile 

Method 

Participants 

The  three  SMEs  from  Study  1  also  participated  in 
this  activity. 

Materials 

A  convenience  sample  of  29  OEs  was  used,  which  did 
not  include  the  6  OEs  used  in  Study  1 .  All  of  the  OEs 
came  from  one  ARTCC.  These  were  selected  because  both 
the  final  report  (FAA  Form  7210-3)  and  the  SATORI 
re-creation  were  available  for  each  in  an  existing  archive. 
A  voice  transcript  was  developed  for  each  OE  from  its 
SATORI  re-creation  so  that  the  SMEs  could  use  a  hard 
copy  of  the  voice  recording.  We  provided  each  SME  with 
copies  of  the  TM  framework,  the  final  OE  reports  and 
the  voice  transcripts  of  each  OE. 


Table  3.  TMTS  from  (TM#2)  the  time  the  relieving  controller  assumed  position  to  — 


Average 

Median 

Min 

Max 

a) 

(TM#9  la)  Time  of  1st  ATC  instruction  by  receiving 
controller  to  A/Cl. 

0:23:51 

0:20:44 

0:00:38 

1:17:25 

b) 

(TM#10)  Time  of  first  point  where  positive  separation 
between  A/C  was  not  ensured. 

0:25:23 

0:21:14 

0:00:38 

1:17:25 

c) 

(TM#1  la)  Time  of  the  last  instruction  by  receiving 
controller  to  A/Cl  prior  to  loss  of  separation. 

0:26:59 

0:21:41 

0:00:38 

1:27:30 

d) 

(TM#14)  Time  separation  lost. 

0:28:30 

0:22:38 

0:01:38 

1:27:32 

Procedure 

The  SMEs  were  instructed  to  fill  out  the  TM  frame¬ 
work  for  each  OE  using  available  information.  The  SMEs 
denoted  aircraft  identified  from  the  OE  materials  as  A/C  1 , 
A/C2,  etc.  according  to  the  procedure  described  previ¬ 
ously.  The  SMEs’  task  was  to  identify  from  the  available 
materials  the  specific  times  for  each  TM  for  the  aircraft 
that  lost  separation.  If  the  TM  was  the  type  that  recurred 
(time  of  1st  ATC  instruction  by  receiving  controller  to 
A/C  1 ,  A/ C2,  etc.) ,  then  they  identified  the  times  for  each 
recurrence  by  using  the  notation  method  developed  in 
Step  1  of  the  project  to  expand  theTM  framework  (i.e., 
TM#9_la,  TM#9_2a,  TM#9_3a  and  so  on)  to  indicate 
temporal  sequence. 

The  SMEs  worked  independently  to  complete  the 
TM  framework  for  each  OE.  When  the  frameworks  were 
completed,  theTM  times  were  compared  for  inconsisten¬ 
cies  between  SMEs.  The  SMEs  were  then  reassembled 
to  resolve  the  discrepancies.  Their  consensus  generated 
the  final  dataset  ofTMs. 

Results 

The  TM  framework  can  be  used  to  temporally  pro¬ 
file  one  OE  (i.e.,  for  a  case  analysis)  or  several  (i.e.,  for 
a  trend  analysis).  The  dataset  ofTMs  identified  by  the 
SMEs  was  used  to  calculate  time  spans  between  TMs 
(TMTSs)  for  each  OE.  Averages  across  OEs  were  then 
calculatedforseveralTMTS  to  determine  if  this  approach 
would  reveal  more  about  OEs  beyond  currently  available 
information. 

Examples  are  shown  in  Table  3  through  Table  6.  In 
each  table,  the  first  time  used  to  calculate  the  TMTSs  is 
the  TM  in  the  table’s  caption.  The  second  time  used  to 
calculate  a  TMTS  is  the  TM  shown  in  the  body  of  the 
table  (i.e.,  a,  b,  c,  d).  The  number  is  the  calculated  time 
spans  between  the  time  of  the  first  TM  and  second  TM 
(in  hours :  minutes :  seconds) .  The  tables  show  the  average 
time  span  and  the  median3  for  the  intervals.  Minimum 
and  maximum  intervals  were  calculated  for  some  examples 
to  determine  if  they  added  additional  information.  The 

3  The  median  is  the  midpoint  value  where  half  the  data  values  are 
above  it  and  half  are  below  it. 


median  value’s  similarity  to  the  average,  minimum  and 
maximum  values  suggests  how  the  value  of  interest  is 
distributed  in  the  sample.  For  example,  in  Table  3,  the 
median  values  of  each  TMTS  are  closer  to  the  average 
values  than  to  either  the  minimum  or  maximum  values. 
However,  Table  4  illustrates  how  the  median  TMTS  can 
be  lower  than  the  average.  This  suggests  that  the  analyst 
should  examine  the  individual  OEs  to  determine  what 
situation  resulted  in  this  data. 

Table  3  shows  some  calculated  time  spans,  beginning 
with  the  time  the  controller  takes  position  responsibil¬ 
ity.  For  example,  row  a)  in  Table  3  describes  when  the 
relieving  controller  first  issued  a  control  instruction  to 
one  of  the  aircraft  that  was  later  involved  in  the  OE.  The 
average  TMTS  (0:23:51)  shows  this  action  was  taken 
within  about  20  min  of  the  controller  taking  over  the 
position.  Row  d)  ofTable  3  shows  that  the  average  time 
interval  from  the  time  the  controller  assumed  position 
responsibility  to  the  time  separation  was  lost  averaged 
about  28  min  (0:28:30)  in  this  sample  of  OEs.  These 
values  are  similar  to  the  national  data  for  the  controller’s 
average  time  on  position  prior  to  an  OE.  However,  the 
intervals  ranged  from  1  min  38  sec  to  1  hr  27  min  and 
32  sec  in  this  sample. 

Table  4  describes  threeTMTS  to  profile  a  communica¬ 
tions  pattern  between  the  controller  and  one  of  the  aircraft 
involved  in  the  OE  (A/Cl).  Row  a)  shows  the  average 
time  between  the  receiving  controller’s  first  instruction  to 
A/Cl  and  the  point  at  which  separation  was  not  ensured 
to  have  been  relatively  short  (0:01:31).  In  some  OEs  in 
this  sample,  separation  was  not  ensured  between  A/Cl 
and  A/C2  prior  to  the  relieving  controller’s  first  commu¬ 
nication  to  A/Cl,  indicated  by  the  negative  value  for  at 
least  one  OE  in  the  sample  (-0:05:29).  Negative  values 
indicate  that  the  second  TM  used  in  the  TMTS  analysis 
preceded  the  first  TM  in  the  OE. 

Table  5  shows  the  average  TMTS  (0:01:30)  from  the 
time  of  the  controller’s  last  instruction  to  A/C  1  until  sepa¬ 
ration  was  lost.  Interestingly,  for  at  least  one  OE  in  this 
sample,  the  interval  from  the  controller’s  last  instruction 
to  the  aircraft  to  the  separation  loss  was  8  min  and  50 
sec,  although  most  TMTS  were  very  short,  as  indicated 
by  the  low  median  value  (0:00:53)  for  this  TMTS. 


Table  4.  TMTS  from  (TM#9_1a)  1st  ATC  instruction  by  receiving  controller  to  A/Cl  to  — 


Average 

Median 

Min 

Max 

a)  (TM#10)  Time  of  first  point  where  positive  separation 
between  A/C  was  not  ensured. 

0:01:31 

0:00:53 

-0:05:29  * 

0:07:38 

b)  (TM#1  la)  Time  of  the  last  instruction  by  receiving 
controller  to  A/Cl  prior  to  loss  of  separation. 

0:03:08 

0:02:23 

0:00:00 

0:10:05 

c)  (TM#14)  Time  separation  lost. 

0:04:38 

0:04:19 

0:00:31 

0:10:07 

*  Note.  The  negative  value  indicates  that  TM#10  preceded  TM#  9  la  in  at  least  one  OE. 

Table  5.  TMTS  from  (TM#1  la)  Time  of  the  last  instruction  by  receiving  controller  to  A/Cl  prior  to  loss  of 
separation  to  — 


Average 

Median 

Min 

Max 

a)  (TM#14)  Time  separation  lost. 

0:01:30 

0:00:53 

0:00:01 

0:08:50 

Table  6.  TMTS  from  (TM#10)  Time  of  first  point  where  positive  separation  between  A/C  was  not  ensured  to  — 


Average 

Median 

Min 

Max 

a)  (TM#1  la)  Time  of  the  last  instruction  by  receiving 
controller  to  A/Cl  prior  to  loss  of  separation. 

0:01:36 

0:00:24 

-0:05:55  * 

0:10:05 

b)  (TM#14)  Time  separation  lost. 

0:03:07 

0:01:59 

0:00:20 

0:10:07 

*  Note.  The  negative  values  indicate  that  the  second  TM  preceded  the  first  TM  in  the  OE. 

Table  7.  TMTS  from  (TM#2)  the  time  the  relieving  controller  assumed  position  to  — 


A/Cl  (n  OEs=29) 

A/C2  (n  OEs=20) 

Average 

Min 

Max 

Average 

Min 

Max 

a)  (TM#9  la)  Time  of  1st  instruction  by 
receiving  controller. 

0:23:51 

0:00:38 

1:17:25 

b)  (TM#10)*  Time  of  1st  point  where  positive 
separation  between  aircraft  is  not  ensured. 

0:25:23 

0:00:38 

1:17:25 

0:25:23 

0:00:38 

1:17:25 

c)  (TM#9  la)  Time  of  1st  instruction  by 
receiving  controller. 

0:28:1 

0:07:44 

1:26:53 

d)  (TM#1  la)  Time  of  the  last  instruction  by 
receiving  controller  prior  to  loss  of 
separation. 

0:26:59 

0:00:38 

1:27:30 

e)  (TM#14)  Time  separation  was  lost. 

0:28:30 

0:01:38 

1:27:32 

0:28:30 

0:01:38 

1:27:32 

f)  (TM#1  la)  Time  of  the  last  instruction  by 
receiving  controller  prior  to  loss  of 
separation. 

0:31:03 

0:09:49 

1:26:53 

Note.  Shaded  rows  indicate  TMs  that  involve  both  aircraft. 
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Table  6  shows  two  TMTS  calculated,  beginning  with 
the  time  when  separation  between  the  aircraft  was  not 
ensured  (i.e.,  if  nothing  intervened  to  change  the  trajectory 
of  either  aircraft,  they  would  eventually  lose  separation). 
As  shown  in  rows  a),  the  average  interval  between  that 
time  and  the  controller’s  last  instruction  to  one  of  the 
aircraft  was  1  min  and  36  sec.  However,  looking  at  the 
minimum  and  maximum  values  shows  that,  in  at  least 
one  OE  in  this  sample,  the  controller’s  last  instruction 
to  one  of  the  aircraft  involved  in  the  OE  occurred  5 
min  and  55  sec  before  separation  was  not  ensured.  At 
the  other  extreme,  in  at  least  one  OE  in  this  sample,  the 
controller’s  last  instruction  to  one  of  the  aircraft  involved 
in  the  OE  occurred  10  min  and  05  sec  after  separation 
was  not  ensured. 

If  data  from  more  OEs  were  available  for  TM  profil¬ 
ing,  other  analyses  could  be  conducted.  For  example, 
OEs  could  be  grouped  by  other  causal  factors  of  interest, 
e.g.,  failure  by  the  controller  to  detect  displayed  data,  the 
controller’s  time  on  position,  combined  positions,  weather 
conditions  and  traffic  complexity  factors.  TMTS  could 
then  be  calculated  for  these  groups  to  determine  whether 
their  TMTS  profiles  differed. 

OE  Profiling 

The  TM  framework  can  be  used  for  case  analysis  to 
understand  the  temporal  evolution  of  one  OE  (e.g.,  pat¬ 
terns  of  communications  between  the  controller  and  the 
aircraft)  or  for  trend  analysis  to  understand  patterns  across 
a  group  of  OEs  (e.g.,  factors  related  to  “dirty”  situations 
handed  off  to  receiving  controllers) .  By  generatingTMTS 
profiles  between  events  that  occurred  during  the  OE  rela¬ 
tive  to  the  interaction  of  the  controller  and  each  aircraft 
(A/Cl,  A/C2,  A/C3,  etc.),  the  results  can  be  compared 
side-by-side  as  in  Table  7,  which  shows  average  TMTSs 
between  some  of  the  TMs  in  this  set.  For  example,  the 
controller’s  first  instruction  to  A/Cl,  on  average,  was 
0:23:51  min  after  taking  position  but  before  the  first 
point  where  no  positive  separation  was  ensured,  which 
occurred  0:25:23  min  after  taking  position.  A/C2  was 
given  its  first  control  instruction,  on  average,  approxi¬ 
mately  5  min  later,  which  occurred  0:28:18  min  after 
taking  position  but  after  the  first  point  where  positive 
separation  was  not  ensured. 

The  controller’s  last  instruction  to  A/Cl,  on  average, 
was  0:26:59  min  after  taking  position  and  before  sepa¬ 
ration  was  lost  (at  0:28:30  min  after  taking  position). 
The  controller’s  last  instruction  to  A/C2,  on  average, 
was  0:31:03  min  after  taking  position.  However  the  last 
instruction  “prior”  cannot  also  occur  “after. ’’This  is  logi¬ 
cally  inconsistent  and  is  a  statistical  artifact  of  using  small 
samples.  Analysts  should  look  instead  at  other  statistics 


such  as  the  median  value  or  mode.4  This  suggests  that 
TMs  should  either  be  used  for  case  analysis  of  individual 
traffic  situations  or  with  large  samples.  Future  work  will 
tease  apart  the  aspects  of  this  phenomenon  as  it  pertains 
to  OE  profiling. 

Summary  of  Step  2 

In  this  activity,  we  demonstrated  how  time  intervals 
calculated  fromTMs  could  be  used  to  profile  the  temporal 
evolution  of  an  OE.  The  project  relied  on  a  convenience 
set  of  archival  OE  reports.  Issues  of  missing  data  and 
small  sample  sizes  related  to  using  archival  OE  materi¬ 
als  were  identified  as  well  as  future  projects  for  this  line 
of  research. 

Project  Summary 

We  identified  a  set  ofTMs  that  could  potentially  occur 
in  every  OE.  Moreover,  all  but  one  TM  can  be  identified 
using  sources  of  objective  data  available  at  the  AT  facil¬ 
ity.  Subjective  assessments  by  the  facility’s  staff  are  not 
needed  to  use  this  method  of  OE  analysis. 

In  Step  1  of  this  project,  we  developed  a  framework 
ofTMs  to  organize  types  of  objective,  time-based  infor¬ 
mation  about  OEs  that  are  routinely  available  at  ATC 
en  route  facilities  but  which  have  not  previously  been 
systematically  organized  in  this  manner. 

In  Step  2,  we  tested  the  framework  developed  in 
Step  1  by  using  it  to  organize  information  from  a  small 
convenience  sample  of  archival  reports  of  en  route  OEs. 
TMs  were  identified  for  each  OE  and  several  TMTSs 
were  calculated  to  demonstrate  the  utility  of  the  meth¬ 
odology.  Results  demonstrated  how  intervals  between 
TMs  could  provide  additional  information  about  how 
OEs  evolve  over  time.  Which  two  TMs  are  selected  to 
calculate  a  TMTS  depends  on  which  dimensions  of  the 
OE  interest  the  analyst. 

Any  trends  in  TMs  and  TMTSs  which  appear  in  OEs 
could  help  the  OE  analyst  to  detect  potential  system 
vulnerabilities.  For  example,  one  might  want  to  identify 
trends  in  the  following  TMs  and  TMTSs. 

Temporal  Markers,  such  as: 

•  the  point  when  the  two  aircraft  were  first  set  up  on  a 
conflicting  course  (that  is,  separation  was  not  ensured 
if  no  one  interceded  and  intervening  actions  would 
be  required  to  maintain  separation). 

•  the  time  when  the  receiving  controller  established 
communication  with  the  aircraft. 


4  The  mode  is  the  value  in  the  sample  that  occurred  most 
frequently. 
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Time  Spans,  such  as: 

•  the  time  interval  from  when  the  controller  assumed 
the  position  relative  to  when  separation  was  not 
ensured. 

•  the  time  interval  between  when  separation  was  not 
ensured  until  separation  was  lost. 

•  the  time  interval  between  the  onset  of  evasive  actions 
and  when  separation  was  lost. 

While  this  project  was  conducted  to  explore  the 
possibility  of  using  objective,  temporal  data  to  better 
understand  OEs,  several  interesting  questions  emerged 
from  this  small  sample.  One  trend  worth  exploring  is 
whether  there  is  a  high  frequency  of  situations  (a)  where 
controllers  “set  up”  an  OE  for  the  next  position  by  hand¬ 
ing  off  a  “dirty”  situation  to  the  receiving  controller  or  (b) 
whether  most  OEs  are  received  “clean”  but  the  receiving 
controller  then  makes  it  “dirty”  by  either  ensuing  actions 
or  inactions.  Finding  that  aircraft  are  handed  off  with 
confliction  could  have  several  implications  for  mitigation 
and  reduction.  If  theTM  profile  of  an  OE  shows  that  the 
first  point  where  aircraft  were  set  on  a  conflicting  course 
occurs  at  a  particular  time  (such  as  prior  to  the  handoff, 
within  the  first  5  min  of  the  controller’s  time  on  position, 
prior  to  establishing  communications  with  the  aircraft), 
the  circumstances  at  that  time  could  be  examined. 

Analysts  could  examine  other  interesting  issues  -  for 
instance,  whether  aircraft  involved  in  OEs  have  other 
temporal  characteristics  in  common,  such  as  time  from 
checking  on  frequency  to  first  control  instruction,  dif¬ 
ferences  in  profiles  for  OEs  occurring  during  the  first 
few  minutes  after  a  position  transfer  or  OEs  with  similar 
causal  factors. 

In  sum,  using  TMs  andTMTSs  to  analyze  OEs  would 
permit  analysts  to  address  several  interesting  types  of 
questions,  such  as  the  following. 

•  Handoff  situations:  Did  the  responsible  controller  set 
up  the  conflict  or  inherit  it?  The  controller  handing 
off  an  aircraft  must  resolve  any  conflicts  prior  to  the 
handoff  to  the  receiving  controller. 

•  Recovery  strategies:  Did  the  aircraft  come  to  the  receiv¬ 
ing  controller  already  in  a  confliction  pattern  (“dirty”) 
so  that  the  controller  had  to  become  a  “fixer?”  If  the 
conflict  was  set  up  before  the  receiving  controller  ac¬ 
cepted  the  aircraft,  strategies  of  the  receiving  controller 
to  fix  separation  could  be  identified,  contributing  con¬ 
ditions  could  be  identified,  handoff  procedures  could 
be  reviewed,  performance  of  the  handoff  controller 
could  be  evaluated,  etc.. 

•  Performance  and  other  factors:  Did  the  aircraft  situa¬ 
tion  come  to  the  controller  “clean”  and  the  situation 
became  “dirty”  from  either  actions  or  inactions  by  the 
controller?  If  the  conflict  was  set  up  after  the  controller 


accepted  the  aircraft,  then  the  receiving  controller  did 
not  ensure  separation  and  the  reasons  for  that  situation 
could  be  examined. 

•  Controller-pilot  factors:  When  was  controller-pilot 
communication  established,  if  at  all?  TM  profiles 
might  show  that  aircraft  involved  in  OEs  were  more 
(or  less)  likely  to  have  communications  established 
between  controller  and  pilot. 

•  Communication  loads:  What  was  the  pattern  of  con¬ 
troller-pilot  communications  before,  during  and  after 
the  OE?  For  example,  the  number  of  communications 
between  the  controller  and  the  pilot(s)  might  be  an 
indicator  of  workload  or  of  controller  efficiency.  TM 
profiles  might  show  that  OEs  differ  from  non-OE 
situations  by  having  a  higher  (or  lower)  rate  of  pilot- 
controller  communications. 

This  approach  to  analysis  could  be  extended  to  a  sys¬ 
tem-wide  application.  Other  interesting  questions  and 
temporal  trends  are  likely  to  be  identified  as  larger  data 
samples  are  analyzed.  These  trends  may  provide  guidance 
for  the  display  of  additional  information  and  formatting 
and  may  have  implications  for  the  design  of  enhanced 
decision  support  capability. 

Limitations 

Convenience  sample 

As  noted  earlier,  the  examples  in  this  paper  were  based 
on  a  set  of  TMs  from  a  small  sample  of  ARTCC  OEs. 
Inferences  from  small  samples  should  be  made  carefully 
until  the  method  can  be  tested  on  a  larger  set  of  OEs. 
For  example,  the  average  times  calculated  from  a  small 
number  of  cases  makes  generalization  beyond  the  sample 
to  other  OEs  difficult  without  verifying  the  results  with 
a  separate  sample  of  OEs.  Other  measures,  such  as  the 
median  (midpoint)  value,  should  also  be  examined  to 
adjust  for  effects  of  outliers  (i.e.,  extraordinary  minimum 
or  maximum  values)  on  the  average  value. 

The  small  sample  constrained  us  to  look  at  the  TMs 
and  TMTSs  only  in  a  general  way.  Much  more  informa¬ 
tion  would  be  available  to  identify  meaningful  trends  if  a 
larger  set  of  OEs  was  analyzed  using  for  this  type  of  profile 
analysis.  Trends  evident  in  larger  samples  of  OEs  would  be 
considered  more  robust,  leading  to  greater  confidence  in 
mitigation  strategies  based  on  the  results.  A  larger  set  of 
OEs  would  permit  a  better  understanding  of  whether  it 
makes  sense  to  average  over  OEs  without  grouping  them 
based  on  other  similarities,  such  as  causal  factors. 

The  ARTCC  operational  errors  we  used  were  selected 
because  the  event  chronology  and  a  SATORI  re-creation 
were  available  for  each  OE.  However,  the  SMEs  judged 
that  the  list  would  also  apply  to  OEs  in  Terminal  Radar 
Approach  Control  facilities,  since  they  are  also  radar 
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environments,  and  possibly  to  OEs  at  air  traffic  tower 
facilities.  This  was  not  verified  in  this  study  in  part  because 
OE  re-creations  were  not  available. 

Missing  data 

In  this  sample,  data  (and  consequently,  perhaps  an 
important  TM)  could  be  missing  for  different  reasons. 
The  temporal  event  might  have  occurred  during  the  OE 
but  the  original  report  omitted  the  necessary  information, 
and  it  was  not  available  from  the  SATORI  for  the  event. 
Thus,  the  corresponding  TM  could  not  be  identified 
later,  resulting  in  missing  data  for  that  TM  in  that  OE. 
Alternatively,  a  temporal  event  might  not  have  occurred 
during  the  OE,  resulting  in  missing  data  for  that  TM  in 
that  OE.  In  our  sample,  we  cannot  distinguish  between 
these  two  types  of  missing  data.  Issues  of  missing  data 
are  not  typically  considered  when  researchers  analyze  OE 
data,  but  when  relying  on  archival  data  only  those  TMs 
can  be  identified  for  which  information  is  available. 

Future  Work 

Other  event  types 

Because  we  wanted  to  use  events  in  OEs  that  could  be 
identified  objectively  as  temporal  markers,  the  result  was 
that  most  TMs  focused  on  the  interactions  between  the 
controller  and  aircraft  or  between  two  or  more  aircraft. 
Future  work  to  test  the  TM  set  will  determine  whether 
other  types  of  events  should  be  included  as  TMs  -  expe¬ 
dited  maneuvers,  altitude  changes,  heading  changes  and 
so  on.  Our  definition  of  temporal  markers  constrains  this 
possibility  somewhat  because  any  TM  candidate  must 
satisfy  the  general  condition  that  these  events  may  differ 
at  the  point  in  time  they  occur,  but  they  occur  in  some 
form  in  most,  if  not  all,  OEs  and  must  be  “objective  or 
calculable.”  This  is  an  important  characteristic  to  enable 
comparison  between  OEs. 

Large  samples 

To  fully  understand  how  to  best  exploit  this  method  of 
analysis,  it  needs  to  be  applied  to  a  large  set  of  OEs.  For 
example,  a  large  ATC  facility  could  apply  this  Temporal 
Marker  framework  to  all  of  its  OEs  for  a  year  and  profile 
OEs  both  individually  and  overall  for  both  case  and  trend 
analysis.  We  could  then  gain  a  better  understanding  of 
OE  profiling  and  be  better  able  to  evaluate  whether 
controllers  and  management  gained  additional  insights 
about  OEs  from  this  type  of  information. 

National  trends 

Currently,  the  chronologies  developed  at  facilities 
after  OEs  are  not  required  to  be  as  detailed  as  the  TM 
framework  nor  are  they  as  highly  structured.  The  TM 
framework  could  help  to  structure  the  OE  chronologies 


if  it  were  used  as  a  tool  to  complete  the  final  OE  report. 
This  would  provide  more  complete  archival  information 
for  OE  analysis.  A  national  database  of  TMs  could  be 
analyzed  for  trends  in  TMs  and  TMTSs  to  expand  our 
knowledge  about  how  OEs  evolve  so  that  vulnerable 
situations  could  be  identified  and  mitigated. 

Re-creations 

To  exercise  this  approach  to  structuring  obj  ective  events 
in  OEs,  a  comprehensive  re-creation  capability  is  needed 
that  would  provide  seamless  visualization  of  traffic  over 
time  and  across  facility  types.  Currently  the  ARTCCs 
have  the  SATORI  system,  TRACONs  have  RAPTOR 
replays  and  some  larger  towers  have  the  potential  to  replay 
traffic  using  the  Airport  Movement  Area  Safety  System 
(AMASS).  These  are  separate  systems  and  have  different 
levels  of  usability  and  timeliness  of  turnaround.  Tower 
facilities  would  present  the  greatest  challenge  for  this 
approach  because  they  currently  have  limited  capability 
to  re-create  traffic  situations.  An  end-to-end  integrated 
capability  would  enable  a  larger  temporal  understanding 
of  the  evolution  of  OEs  (and  potentially  other  types  of 
events)  that  is  not  currently  available. 

Conclusions 

In  sum,  we  believe  that  using  the  TM  framework  will 
give  operational  personnel  and  researchers  another  tool 
to  identify  information  about  factors  leading  to  OEs. 
By  profiling  OEs  in  this  manner,  this  method  provides 
an  objective  method  for  identifying  structure  in  high 
consequence,  complex,  dynamic  events.  This  approach 
provides  a  means  to  view  OEs  in  a  new  light  that  has  the 
potential  to  directly  lead  to  mitigation  activities  and  to 
ensure  continual  improvement  in  our  understanding  of 
OEs  and  the  OE  analysis  process. 
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